Gesture recognition with sensors

ABSTRACT

A sensor for motion or gesture sensing may be configured to emit radio frequency signals such as for pulsed range gated sensing. The sensor may include a radio frequency transmitter configured to emit the pulses and a receiver configured to receive reflected ones of the emitted radio frequency signals. The received pulses may be processed by a motion channel and/or a gesture channel. The gesture channel may produce signals for further processing for identification of one or more different motion gestures such as by calculating and evaluating features from one or more of the amplitude, phase and frequency of the output signals of the gesture channel. The sensing apparatus may optionally serve as a monitor for evaluating user activities, such as by counting certain activities. The sensor may optionally serve as a user control interface for many different devices by generating control signal(s) based on the identification of one or more different motion gestures.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a national phase entry under 35 U.S.C. § 371of International Application No. PCT/EP2016/058806 filed Apr. 20, 2016,published in English, which claims priority from U.S. Provisional PatentApplication No. 62/150,086 filed Apr. 20, 2015, all of which areincorporated herein by reference.

FIELD OF THE TECHNOLOGY

The present technology relates to methods and apparatus for detection ofcharacteristics of moving objects and living subjects. Moreparticularly, it relates to sensing or recognizing gestures or otherbodily motions such as with radio frequency sensors.

BACKGROUND OF THE TECHNOLOGY

Continuous wave (CW) Doppler radar motion sensors emit a continuous waveradio frequency (RF) carrier and mix the transmitted RF with the returnechoes to produce a difference frequency equal to the Doppler shiftproduced by a moving target. These sensors do not have a definite rangelimit (i.e., they can receive signals for both near and far objects,with the received signal being a function of radar cross section). Thiscan lead to false triggers i.e., motion artefact interference. They mayalso have an undesirably high sensitivity at close range that leads tofalse triggering.

A pulse Doppler motion sensor is described in U.S. Pat. No. 4,197,537 toFollen et al. A short pulse is transmitted and its echo is self-mixedwith the transmitted pulse. The pulse width defines the range-gatedregion. When the transmit pulse ends, mixing ends and target returnsarriving after the end of the transmit pulse are not mixed and arethereby gated out.

A Differential pulse Doppler motion sensor disclosed in U.S. Pat. No.5,966,090, “Differential Pulse Radar Motion Sensor,” to McEwan,alternately transmits two pulse widths. It then subtracts the Dopplerresponses from each width to produce a range gated Doppler sensingregion having a fairly constant response versus range.

Impulse radar, such as that described in U.S. Pat. No. 5,361,070,“Ultra-Wideband Radar Motion Sensor,” to McEwan produces a very narrowsensing region that is related to the transmitted impulse width. Atwo-pulse Doppler radar motion sensor, as described in U.S. Pat. No.5,682,164, “Pulse Homodyne Field Disturbance Sensor,” to McEwan,transmits a first pulse and after a delay generates a second pulse thatmixes with echoes from the first pulse. Thus a range gated sensing bandis formed with defined minimum and maximum ranges. UWB radar motionsensors have the disadvantage of not having global RF regulatoryacceptance as an intentional radiator. They also have difficulty sensingobjects at medium ranges and in some embodiments can be prone to RFinterference.

A modulated pulse Doppler sensor is described in U.S. Pat. No. 6,426,716to McEwan. The range gated microwave motion sensor includes adjustableminimum and maximum detection ranges. The apparatus includes an RFoscillator with associated pulse generating and delay elements toproduce the transmit and mixer pulses, a single transmit (TX)/receive(RX) antenna or a pair of separate TX and RX antennas, and an RFreceiver, including a detector/mixer with associated filtering,amplifying and demodulating elements to produce a range gated Dopplersignal from the mixer and echo pulses.

In U.S. Pat. No. 7,952,515, McEwan discloses a particular holographicradar. It adds a range gate to holographic radar to limit response to aspecific downrange region. McEwan states that cleaner, more clutter-freeradar holograms of an imaged surface can be obtained, particularly whenpenetrating materials to image interior image planes, or slices. Therange-gating enables stacked hologram technology, where multiple imagedsurfaces can be stacked in the downrange direction.

In U.S. Patent Application Publ. no. 2010/0214158, McEwan discloses anRF magnitude sampler for holographic radar. McEwan describes that the RFmagnitude sampler can finely resolve interferometric patterns producedby narrowband holographic pulse radar.

In U.S. Patent Application Publication No. 2014/0024917, McMahon et al,describe a sensor for physiology sensing that may be configured togenerate oscillation signals for emitting radio frequency pulses forrange gated sensing. The sensor may include a radio frequencytransmitter configured to emit the pulses and a receiver configured toreceive reflected ones of the emitted radio frequency pulses. Thereceived pulses may be processed to detect physiology characteristicssuch as motion, sleep, respiration and/or heartbeat.

There may be a need to improve sensors and processing for radiofrequency sensing such as for detection or recognition of particularmotions or gestures.

SUMMARY OF THE TECHNOLOGY

One aspect of some embodiments of the present technology relates to asensor for detecting gestures or particular bodily motions.

An aspect of some embodiments of the present technology relates to asensor for detecting a gesture or bodily motion with radio frequencysignals.

Another aspect of some embodiments of the technology relates to such asensor with a circuit configured to generate pulsed radio frequency (RF)signals such as for gesture recognition or bodily movement typerecognition.

Some versions of the present technology may include a radio frequencymotion sensing apparatus. The apparatus may include a radio frequencytransmitter configured to emit radio frequency signals. The apparatusmay include a receiver configured to receive reflected ones of theemitted radio frequency signals. The apparatus may include a motionchannel circuit configured to process the received reflected ones of theemitted radio frequency signals and produce motion output signals. Theapparatus may include a processor configured to evaluate the motionoutput signals and identify a motion based on any one or more ofamplitude, phase and frequency of the motion output signals.

In some cases, the identified motion may include at least one of handgesture, an arm gesture or a combined hand and arm gesture. Theidentified motion may include a rollover motion. The identified motionmay include an activity. The identified motion may include a shavingactivity.

In some cases, the motion output signals may include in phase andquadrature phase signals. The emitted radio frequency signals mayinclude pulsed radio frequency oscillating signals. The motion channelcircuit may include a bandpass filter. The apparatus, or the processingcircuits thereof, may be configured to demodulate the received reflectedones of the emitted radio frequency signals with signals representingthe emitted radio frequency signals. The apparatus, or the processingcircuits thereof, may be configured to calculate time difference and/orphase difference between the emitted radio frequency signals and thereceived reflected ones of the emitted radio frequency signals and toidentify the motion based on the calculated time difference and/or phasedifference.

In some cases, the motion channel may include an antialiasing filter.The processor may be configured to classify a motion based on aplurality of features calculated from any two of the amplitude, phaseand frequency of the motion output signals. The processor may beconfigured to classify or identify a motion based on a durationcalculated with any one or more of the amplitude, phase and frequency ofthe motion output signals. The processor may be configured to calculatethe plurality of features from each of the amplitude, phase andfrequency of the motion output signals. The plurality of features mayinclude a determined duration derived from analysis of any one or moreof the amplitude, phase and frequency of the motion outputs signals.

In some cases, the calculated plurality of features may include one ormore of:

(a) a frequency characteristic derived from stopped frequency through agesture in motion up to some maximum frequency, then back to stoppedagain;

(b) a time and frequency analysis of the signal comprising any of shorttime Fourier transform, peak and harmonic tracking and/or channelprocessing of an I and/or Q channel(s);

(b) a phase characteristic comprising any of: a phase difference betweenI and Q signals and an evaluation of a repetitive signal within acertain number of standard deviations of a mean of characteristicchange;

(c) an amplitude characteristic comprising any of: peak and troughdetection, zero crossing detection, and envelope of signal detection;and

(d) a learn skewness, kurtosis, spread in frequency, phase, amplitude,mean, and/or standard deviation.

Optionally, the processor may be configured to compare the calculatedplurality of features to one or more thresholds. The processor may beconfigured to identify the motion by selecting one from a plurality ofpredetermined motions. The processor may be configured to count a numberof occurrences of the identified motion. The processor may be furtherconfigured to generate a control signal for operation of a device basedon the identified motion. The processor may be further configured togenerate different control signals for different operations of a devicebased on different identified motions. The processor may be configuredto evaluate the motion output signals and identify a motion based onmotion output signals from a plurality of sensors. The plurality ofsensors may include a first sensor and a second sensor, wherein theprocessor evaluates I and Q signals from the first sensor and the secondsensor to identify the motion. The processor may determine an Idifferential and a Q differential of the I and Q signals from the firstsensor and the second sensor. The plurality of sensors may include afirst sensor, second sensor and a third sensor, wherein the processorevaluates I and Q signals from the first sensor, the second sensor andthe third sensor to identify the motion. In some cases of the apparatus,at least two of the three sensors may be positioned to be orthogonal toeach other. The evaluated I and Q signals may be used by the processorto identify movement characteristics in more than one dimension. Theprocessor may be configured to extract one or more of the followingparameters of the moving object: velocity, change in velocity, distance,change in distance, direction and change in direction.

Some versions of the present technology may involve a method for radiofrequency motion sensing. The method may include, with a radio frequencytransmitter, emitting radio frequency signals. The method may include,with a receiver, receiving reflected ones of the emitted radio frequencysignals. The method may include processing the received reflected onesof the emitted radio frequency signals to produce motion output signalswith a motion channel circuit. The method may include, in a processor,evaluating the motion output signals and identifying a motion based onany one or more of amplitude, phase and frequency of the motion outputsignals.

In some cases, the identified motion may include any one of handgesture, an arm gesture and a combined hand and arm gesture. Theidentified motion may include a rollover motion. The identified motionmay include an activity. The identified motion may include a shavingactivity. The motion output signals of the method may include in phaseand quadrature phase signals. The emitted radio frequency signals mayinclude pulsed radio frequency oscillating signals. The motion channelcircuit may include a bandpass filter.

In some cases, the method may involve demodulating the receivedreflected ones of the emitted radio frequency signals with signalsrepresenting the emitted radio frequency signals. The method may includecalculating time difference and/or phase difference between the emittedradio frequency signals and the received reflected ones of the radiofrequency signals and identifying the motion based on the calculatedtime difference and/or phase difference. The motion channel may includean antialiasing filter.

In some cases, the method may include, with or in the processor,classifying a motion based on a plurality of features calculated fromany two of the amplitude, phase and frequency of the motion outputsignals. The processor may classify or identify a motion based on aduration calculated with any one or more of the amplitude, phase andfrequency of the motion output signals. The method may involvecalculating, in the processor, the plurality of features from each ofthe amplitude, phase and frequency of the motion output signals. Theplurality of features may include a determined duration derived fromanalysis of any one or more of the amplitude, phase and frequency of themotion outputs signals.

In some cases, the calculated plurality of features may include one ormore of:

(a) a frequency characteristic derived from stopped frequency through agesture in motion up to some maximum frequency, then back to stoppedagain;

(b) a time and frequency analysis of the signal including any of a shorttime Fourier transform, peak and harmonic tracking, and processing of Iand/or Q channel(s);

(b) a phase characteristic including any of a phase difference between Iand Q signals or an evaluation of a repetitive signal within a certainnumber of standard deviations of a mean of characteristic change;

(c) an amplitude characteristic including any of a peak and troughdetection, a zero crossing detection, an envelope of signal detection;and

(d) a learn skewness, kurtosis, spread in frequency, phase, amplitude,mean, and/or standard deviation.

Optionally, the method may involve, in the processor, comparing thecalculated features to one or more thresholds. The method may involve,in the processor, identifying the motion by selecting one from aplurality of predetermined motions. The method may involve, in theprocessor, counting a number of occurrences of the identified motion.The method may involve, with the processor, generating a control signalfor operation of a device based on the identified motion. The method mayinvolve, with the processor, generating different control signals fordifferent operations of a device based on different identified motions.

In some cases, the processor may evaluate the motion output signals froma plurality of sensors and identifies a motion based on the evaluatedmotion output signals. The plurality of sensors may include a firstsensor and a second sensor, and the processor may evaluate I and Qsignals from the first sensor and the second sensor to identify themotion. The processor may determine an I differential and a Qdifferential of the I and Q signals from the first sensor and the secondsensor. Optionally, the plurality of sensors may include a first sensor,a second sensor and a third sensor, and the processor may evaluate I andQ signals from the first sensor, the second sensor and the third sensorto identify the motion. In some cases, at least two of the three sensorsmay be positioned to be orthogonal to each other. The evaluated I and Qsignals may be used by the processor to identify movementcharacteristics in more than one dimension.

Other aspects, features, and advantages of this technology will beapparent from the following detailed description when taken inconjunction with the accompanying drawings, which are a part of thisdisclosure and which illustrate, by way of example, principles of thetechnology. Yet further aspects of the technology will be apparent fromthe appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Further example embodiments of the technology will now be described withreference to the accompanying drawings, in which:

FIG. 1 is an illustration of an example detection apparatus suitable forimplementation with a radio frequency physiology sensor of the presenttechnology;

FIG. 2 is a diagram illustrating a conceptual structure and process flowfor evaluation of sensor signals suitable for some embodiments of thetechnology;

FIG. 3 is a block diagram with processing for a sensor apparatus in someversions of the technology;

FIG. 4 illustrates phase response of an example gesture transferfunction suitable for the present technology;

FIG. 5A is a graph showing a movement channel output magnitude overdistance for a sensor;

FIG. 5B contains graphs comparing a movement channel and gesture channelwith respect to output magnitude over distance for another sensor;

FIG. 6A illustrates a frequency response of a gesture channels for aversion of the sensor;

FIG. 7 illustrates a motion gesture with respect to an example sensor;

FIGS. 8 and 9 illustrate amplitude, frequency and phases responses for asingle motion gesture of FIG. 7 and the repeated motion gesturerespectively;

FIG. 10 is another plot of the phase response associated with the motiongesture of FIG. 7;

FIG. 11 illustrates another motion gesture with respect to an examplesensor;

FIG. 12 illustrates amplitude, frequency and phases responses for therepeated motion gesture of FIG. 11;

FIG. 13 illustrates detection of change in motion direction from I and Qsignal difference;

FIG. 14 illustrates a counting activity stroke methodology, such as withzero crossing detection;

FIG. 15 illustrates a training processing methodology for activitydetection involving block based classification;

FIG. 16 illustrates block by block classification of an gesture activitysignal;

FIG. 17 show signal graphs for stroke detection and stroke ratedetection;

FIGS. 18A-C illustrate a motion gesture and the amplitude, phase andfrequency response by a suitable sensor to its repetition and the singlemovement;

FIGS. 19A-C illustrate another motion gesture and the amplitude, phaseand frequency response by a suitable sensor to its repetition and thesingle movement;

FIGS. 20A-C illustrate a further motion gesture and the amplitude, phaseand frequency response by a suitable sensor to its repetition and thesingle movement;

FIGS. 21A-C illustrate a still different motion gesture and theamplitude, phase and frequency response by a suitable sensor to itsrepetition and the single movement;

FIGS. 22A-C illustrate yet another different motion gesture and theamplitude, phase and frequency response by a suitable sensor to itsrepetition and the single movement;

FIGS. 23A-C illustrate another motion gesture and the amplitude, phaseand frequency response by a suitable sensor to its repetition and thesingle movement;

FIGS. 24A-C illustrate a still different motion gesture and theamplitude, phase and frequency response by a suitable sensor to itsrepetition and the single movement;

FIGS. 25A-B illustrate a rollover motion and the amplitude, phase andfrequency response by a suitable sensor to its repetition and the singlemovement;

FIGS. 26A-B illustrate another rollover motion and the amplitude, phaseand frequency response by a suitable sensor to its repetition and thesingle movement;

FIGS. 27A-B illustrate a further rollover motion and the amplitude,phase and frequency response by a suitable sensor to its repetition andthe single movement;

FIGS. 28A-B illustrate yet another rollover motion and the amplitude,phase and frequency response by a suitable sensor to its repetition andthe single movement;

FIG. 29A illustrates maximum quadrature (I/Q) phase detection in amulti-sensor (e.g., stereo sensor system) setup. To always get maximumphase the sensors may be placed orthogonal to each other (unless themotion is in a downwards direction—this direction is not depicted in thefigure).

FIG. 29B illustrates another example of a multi-sensor (e.g., stereosensor system setup). The section labelled ‘sweet spot’ on the figuredepicts the area that is optimum focus for both sensors, due to thecurved beam pattern. This shows that the sensors do not need to beorthogonal.

FIG. 30 illustrates processing of image data by splitting the colourdata from a spectrogram into RGB channels and selecting main blob areas.

DETAILED DESCRIPTION

As illustrated in FIG. 1, some embodiments of the present technology mayimplement a sensing or detection apparatus 100, such as one configuredwith particular processing methodologies, useful for detectingparticular motions of a user or a patient (the patient may be identicalor a different person from the user of the detection apparatus 100) inthe vicinity of the apparatus. The sensor may be a standalone sensor ormay be coupled with other apparatus, such as a respiratory treatmentapparatus or sleep assessment apparatus. For example, it may optionallyprovide an automated treatment response based on an analysis of thegestures or motion detected by the sensor of the apparatus. For example,a respiratory treatment apparatus with a controller and a flow generatormay be configured with such a sensor and may be configured to adjust apressure treatment generated at a patient interface (e.g., mask) inresponse to particular motions or gestures detected by the sensor. Therespiratory treatment apparatus may be for example, a respiratorytherapy or PAP apparatus, such as any one described in InternationalPatent Application Publication No. WO 2013/152403, the entire disclosureof which is incorporated herein by reference.

In general, such motions or gestures may be understood to be any thatare intentionally or subconsciously made by a person rather than thosephysiological characteristics that are involuntarily periodic in nature,(i.e., chest movement due to respiration or cardiac activity.) In thisregard, movement signals sensed by a sensor that are generated byparticular human gestures may be processed to identify or characterizethe particular movement or gesture. For example, a hand movement, orparticular hand movement, could be detected.

Larger movements such as the movement made by a person turning over inbed (a turnover) can also be recognized. Particularized detection ofsuch movement events may then permit them to be counted or serve as acontrol for an apparatus (e.g., implemented to turn on or off a system,or to provide other control signals). The technology may also beimplemented to classify physiological movement such as sway, breathing,and faster motion such as shaving or scratching. It could be implementedto improve the robustness of breathing rate detection when a subject isstanding or sitting, such as by identifying and eliminating such swayand gesture motion for respiratory rate detection. The technology may beeven be implemented to monitor subjects with persistent itch, irritationor discomfort, e.g., in a clinical trial of a dermatological cream forquantification of such itch related or discomfort related motion. Insome cases, it could even be implemented to assess the efficacy ofconsumer products dependent on motion such as a shaving blade orcream/gel, and understand shaving motions, etc.

A sensor with suitable processing circuit(s) (e.g., one or moreprocessors) may be configured as a gesture detection apparatus that maybe implemented as a component (e.g., a control component) for manydifferent types of apparatus. For example, a television or televisionreceiver may include such a sensor for controlling the operations of thetelevision or television receiver with different gestures (e.g., on/off,volume changes, channel changes etc.). Similarly, the gesture detectionapparatus may be configured as part of a user interface for a gamingapparatus or computer, such as to control operations of the game orcomputer. Such a gesture detection apparatus may be implemented for manyother apparatus that employ a user interface such that the userinterface may be implemented as a gesture-controlled user interface. Forexample, a processor or controller may evaluate signals from one or moresensors to identify in the processor or controller a particular movementor gesture, and in response, activate generation of a visual (or audio)change to a displayed user interface (such as one displayed on a displaydevice such as an LCD or LED screen). The identified gesture or theactivated change may be used to issue one or more control signals tocontrol a device (e.g., a computer, television, computer game console,user appliance, automated machine, robot, etc., that is coupled to, orcommunicates, with the processor or controller.

A typical sensor, such as a radar sensor, of such an apparatus mayemploy a transmitter to emit radio frequency waves, such as radiofrequency pulses for range gated sensing. A receiver, which mayoptionally be included in a combined device with the transmitter, may beconfigured to receive and process reflected versions of the waves.Signal processing may be employed, such as with a processor of theapparatus that activates the sensor, for gesture or motion recognitionbased on the received reflected signals.

For example, as illustrated in FIG. 2, the transmitter transmits aradio-frequency signal towards a subject, e.g., a human. Generally, thesource of the RF signal is a local oscillator (LO). The reflected signalis then received, amplified and mixed with a portion of the originalsignal, and the output of this mixer may then be filtered. In somecases, the received/reflected signal may be demodulated by thetransmitted signal, or the phase or time difference between them may bedetermined, for example, as described in US-2014-0163343-A1, the entiredisclosure of which is incorporated herein by reference.

The resulting signal may contain information about the movement (e.g.,gestures), respiration and cardiac activity of the person, and isreferred to as the raw motion sensor signal. In some cases, the signalmay be processed to exclude involuntary periodic activity (e.g.,respiration and/or cardiac activity) so that movement information in thesignal may be classified for its particular gesture or movement type. Insome cases, the sensor may be a sensor described in U.S. PatentApplication Publication No. 2014/0024917, the entire disclosure of whichis incorporated herein by reference.

The sensor may include various motion channels for processing ofdetected signals, for example, such a sensor may be implemented with agesture processor to provide a gesture channel output signal. This maybe distinct from a movement processor that provides a movement channeloutput signal. Having multiple processors can permit output of signalswith different characteristics (e.g., different bandwidths, differentsampling rates, etc.) for different motion evaluations. For example,there may be more information in a gesture signal rather than abreathing or cardiac signal. For example, the gesture signal can includeinformation representing detection of a wider range of motion speeds.For example, a 1 metre per second movement might cause a 70 Hz basebandsignal in a 10.525 GHz receiver. A typical sensing scenario might beable to detect speeds of between 1 mm/s to 5 m/s. For gesture detection,frequencies greater than 10 Hz (1 cm/s up to 5 m/sec) may be evaluated.For breathing, detection may involve evaluation of frequenciescorresponding to velocities in range of 1 mm/sec to approximately 1 m/s.Thus, a movement processor may generate a signal focused on slowermovements, and a gesture processor may generate a signal with a muchwider band that may include both slow movements as well as fastermovements. Thus, the sensor may implement analog and/or digital circuitcomponents, for signal processing of the received sensor signal. Thismay optionally be implemented, at least in part, in one or more digitalsignal processors or other application specific integrated chips. Thus,as illustrated in FIG. 3, the sensor may be implemented with the gestureprocessor to implement a particular transfer function (Hg), as well asan additional movement processor to implement a particular transferfunction (Hm), either of which may be considered a motion processor orchannel circuit for producing motion output signals.

For example, in some cases, the sensor may have a gesture channel thatprovides quadrature output signals (I and Q) whose amplitude, frequencyand phase is given by:VI(x,t)=Hg(jω)A(x)Sin(4πx(t)/λ+ϕ)VQ(x,t)=Hg(jω)A(x)Sin(4πx(t)/λ+ϕ+π/2)

Where:

Hg(jω) is the transfer function of the sensor gesture channel such as ina baseband circuit or baseband processor;

A(x) is the demodulated received signal strength and hence dependent ontarget radar cross section (size) and target distance (x);

x(t) is the displacement of the target with time

λ is the wavelength of the RF signal (e.g., a wavelength in free spacecorresponding to a 10.525 GHz frequency signal (e.g., a wavelength of28.5 mm); and

Jω—Is the frequency response of the system where co is the angularvelocity and j is the complex number (0+√−1), which provides the phaseinformation).

The gesture channel will have a frequency response to movement. For anin-band movement signal with a linear velocity v which moves a distancedx from position x0 to position x1 towards or away from the sensor in atime interval dt starting at t0 and ending at t1 the gesture channeloutput signal frequency f is given by2πf(t1−t0)=4π(x1−x0)/λ,2πfdt=4πdx/λ

For a 10.525 GHz, 28.5 mm λ sensor

-   -   f˜70.17 v where f (Hz) and v(m/s)) Here, taking λ into account,        the units match on both sides such that wf (1/s), v (m/s) and        2/λ, =70.175 (m{circumflex over ( )}−1). The constant value of        70 is actually 1/(2λ) and has the dimension of m−1.

In general:f(t)=2v(t)/λTypically, the amplitude of the output signal at any particularfrequency will depend on the gesture channel transfer function frequencyresponse.

The gesture channel will also have a phase response to movement of thetarget (e.g., a person's hand etc.). The phase difference between the Iand Q channels is 90 degrees. As a result the Lissajous curve for the Iand Q signal is a circle, as shown in FIG. 4. The frequency (cycle time)is determined by the target speed. The amplitude is determined by thetarget distance, target cross section and by the gesture channeltransfer function. The direction of the phasor, clockwise oranti-clockwise, is dependent on the direction of the motion towards oraway from the sensor.

The gesture channel or another, general movement dedicated, channel mayalso have an amplitude response to non-gesture related movement. Theamplitude of its I and Q corresponding channels is determined by thetarget distance, target cross section and by the movement channeltransfer function. By way of example, a logarithmic plot of the movementchannel signal amplitude versus target distance for a fixed target andin band target speed is as shown in FIG. 5A.

FIG. 5B compares the magnitude response of two channels (movementchannel and gesture channel) in response to a specific movement overdistance of a different sensor from that of FIG. 5A. The gesture channelhas a similar characteristic to the movement channel. FIG. 5Billustrates channel amplitude response of a version of the sensor, suchas with different antialiasing filtering compared to that of FIG. 5A.Because of the radar equation and associated antenna gain transferfunction, as well as a non-linear scattering of the reflected signal,the receive signal level declines as function of the distance. (e.g.,1/xn, 1.5<n<3).

Accordingly, by processing of the gesture output signal(s) from thegesture channel and/or the movement signals from the movement channel(which may or may not be the same as the gesture channel), particulargestures or movements may be detected in one or more processors. Thismay be accomplished by calculating features from the signal(s) andcomparing the features and/or changes in the features to one or morethresholds, or identifying patterns in the signal(s). Such features ofthe signals may be, for example, statistical values of parametersassociated with the signal(s), such as average or the median values ofthe signal(s) phase, amplitude and/or frequency, standard deviation ofany of these values etc.

Suitable features may be determined by training of a classifier.Classification of calculated features may then serve as a basis forgesture detection with the trained classifier. For example, one or moreprocessors may evaluate any one or more of the gesture signal(s) phase,amplitude and/or frequency characteristics to detect patterns or otherindicia in the signal associated with a particular gesture or movement.In some cases, the characteristics may include amplitude cadence (e.g.,amplitude and sidebands) and a time during which the gesture persists.In this regard, analysis of the signal(s) will permit identification ofsignal characteristics that are produced with respect to certain motions(e.g., towards or away from) in relation to the sensor since differentmotions may produce differently defined amplitude, frequency and/orphase characteristics. Such an analysis may include choosing a patternfor a particular gesture so as to distinguish between several gestures(e.g., select one from a group of different predetermined trainedgestures.) In some cases, the system may also process feedback from theuser based on a perceived correct or incorrect detection of a movementor gesture signal. The system may optionally update its classificationbased on this input, and may optionally prompt the user to perform oneor more repetitions of a specific gesture in order to optimizeperformance/recognition. In this manner, the system may be configured toadapt (personalise) to the gestures of a particular user, and identifyand separate (distinguish) the gestures of different users.

In this regard, fast or slow and/or long or short hand gestures towardsor away from the sensor can produce clearly detectable signals. Motionacross the sensor produces a motion component that is also towards andaway from the sensor, but this motion component is small. Thereforemotion across the sensor produces distinguishing characteristics but atsmaller amplitude, lower frequency and a center line based phase change.

Motion towards the sensor always has a specific phase rotation which isreversed when the motion is away from the sensor. Phase can thereforeprovide gesture directional information. A frequency spectrogram mayclearly show the characteristic motion velocity for particular gesturesand may be identified by processing features of the spectrogram.

The amplitude characteristic may require signal conditioning before use,as the amplitude is seen to vary with position (distance from thesensor) as well as target cross section/size.

It is possible to extract the radial velocity and direction of a target.Within the sensor range (e.g. 1.8-2 m), it might be a small target nearin or a larger target further away. Thus, a processor of the any one ormore of velocity, change in velocity, distance, change in distance,direction, change in direction, etc., extracted from the gesture channelmay also serve as characteristics for detection of particular gestures.

In general, the frequency and amplitude of the signals output from thegesture and movement channels are dependent on the baseband circuitamplification and filtering. In one version, the circuit implementingthe gesture/movement transfer function may be constructed with a bandpass filtered amplifier with a gain, (e.g., 9.5) and with a frequency BW(bandwidth) (e.g., approximately 160 Hz) in a desired range (e.g.,approximately 0.86 Hz to 161 Hz). Such an example is illustrated in thetransfer function simulation graph of FIG. 6A. This may optionally beimplemented with both low pass and high pass filters.

In some versions, the gesture channel may include an antialiasingfilter. The gesture channel frequency characteristics may includegreater or lesser antialiasing filtering. As shown in this particularexample, there is less than 10% drop in signal level (6.7 to 6.1 drop ingain) at the band edge of 160 Hz. In some cases, the antialiasingfiltering may be implemented by the band pass filter described in theabove paragraph.

In some cases, a processor may calculate time difference or phasedifference between the emitted and the received signals of the sensorand identify particular motions/gestures based on the calculated timedifference and/or phase difference.

In the following, example gesture detection is described in reference tocertain gestures/motions, such as hand and/or arm movement that may betrained in a system, such as for detection with a classifier executed bya processor. Other gestures may also be trained.

For example, in some versions, a group of processing methodologies(e.g., algorithm processing steps) and associated digital signalprocessing may be implemented for determining physiological repetitiveand/or varying motion, including that caused by the movement of chestdue to respiration, sway detection and cancellation, and gross and finemovement detection (gesture detection) due to a multitude of actionssuch movement of the hands and arms, shaving (e.g., of the face) orscratching (e.g., due to physical irritation or discomfort). The keyinput features to such a system are derived from any one or more ofamplitude (temporal), frequency and phase characteristics of thedetected signal.

In essence, the processing applied allows the unraveling of thedirection change information from the in phase (I) and quadrature phase(Q) signals in the presence of significant noise and confoundingcomponents (due to the sensor's inherent noise, sensor signal“fold-over” (dependent on frequency), sensor phase imbalance (ifpresent), different type of physiological movement, and other motionsources and background clutter). The processed channel signals (in phaseand quadrature) may be recorded by a radio frequency RADAR and may bedigitised using a suitable ADC module. These RF signals can becontinuous wave, pulsed (e.g., applied to 10.525 GHz sensor or others)or pulsed continuous wave.

The signals may be fed or input into a filter bank, where a series ofdigital filters including bandpass filtering are applied to detect andremove low frequency sway information.

The phase information in the two channels may be compared to produce aclockwise/anti-clockwise pattern. Hysteresis and glitch detection may beapplied to suppress signal fold-over, and the resulting signalrepresents the relative direction of the movement source to the sensorframe of reference. Peak/trough detection and signal following may beadditionally implemented to aid this processing. Therefore, the systemcan determine if a movement is directed towards or away from the sensor,and if changing direction.

The analog filtering on the sensor can be modified to widen thebandwidth prior to sampling in some versions.

Example Gestures/Movements

Gesture A:

Detectable Gesture A may be considered in reference to FIGS. 7-10. Inexample A, the gesture is based on hand movement, such as when a personsits approximately 70 cm in front of the sensor. The movement beginswith the palm of the hand (face up or forward facing the sensor)approximately 40 cm from the sensor. This was the furthest point duringthe gross motion. The closest point may be approximately 15 cm. The handis extended (moves) towards the sensor (taking 1 second) and after abrief pause is pulled back (taking 1 second). The movement may beconsidered as a replication of a sine wave. The complete gesture takesapproximately 2 seconds. Sensor recordings from the gesture channel areshown with respect to repetition of the single gesture (10 times in FIG.8 and a single time in FIG. 9). FIG. 8 illustrates a plot of changingamplitude verses time, frequency (spectrogram) and changing phase datawith respect to time of the sensor recordings from the gesture channel.The phase direction may be plotted with respect to time by applying theI and Q signal outputs to different axis as illustrated in FIGS. 8, 9and 10.

FIGS. 8-10 show in reference to the gesture A that motion towards thesensor has a specific phase rotation which is reversed when the motionis away from the sensor. Thus, analysis of this phase can providegesture directional information.

The frequency spectrogram clearly shows the characteristic motionvelocity for the gesture. This frequency “chirp” has a distinctpersonality (i.e., can be classified in a processor). FIG. 9 depicts aclose-up view of the motion/gesture outlined in FIG. 7. FIG. 8 depictsmultiple instances of this gesture; the time domain signal amplitude isshown, as well as a spectrogram, and a phase plot. The spectrogramindicates time on the x-axis, frequency on the y-axis, and intensity ata particular time for a particular frequency as a different colour. Inthis example, the subject sat approximately 70 cm in front of thesensor. The movement begins with the palm of the hand (face up) 40 cmfrom the sensor, the furthest point during the gross motion. The closestpoint was 15 cm. The hand is extended towards the sensor (1 second) andafter a brief pause is pulled back (1 second). The intention was toreplicate a sine wave. The complete gesture took 2 seconds. The completegesture was repeated 10 times (as per FIG. 8). FIG. 9 indicates wherethe movement is towards the sensor, close to the sensor, then movingaway from the sensor. For this case, the maximum frequency is seen torange from 90-100 Hz. The phase is seen to move clockwise during motiontowards the sensor, and anti-clockwise when moving away. In FIG. 10, theI and Q (in phase and quadrature) channels were plotted against time ona 3D figure using MATLAB (The Mathworks, Natick) as the second method ofanalysis for phase direction.

The amplitude characteristic may employ signal conditioning before use,as the amplitude is seen to vary with position (distance from thesensor) as well as target cross section/size.

The radial velocity and direction of a target may also be extracted.Within the sensor range (e.g., 2 m), it (the target) might be a smalltarget near in or a larger target further away.

Gesture B

Another detectable gesture B (arm and hand) may be considered inreference to FIGS. 11-12. Movement begins with the arm fully extended.As shown in FIG. 11, a hand is then swung completely across the body.The palm naturally changes from face up to face down as the arm is movedfrom close to the sensor (5 cm) to furthest away from the sensor (135cm). At the midway point of the gesture (at the peak of the arm swingarc over the head) the palm direction will change.

The complete gesture takes less than 4 seconds and may, for example, beperformed in a sitting position.

The gesture B with an approximately 2 m/s velocity produces a frequencyof 140 Hz. This occurs within a 1 m distance over a 1 second period witha start and end velocity of 0 m/s.

The sensor may be positioned approximately near the person for detection(e.g., 95 cm from the center of the chest). For example, a furthestpoint during the gross motion of the gesture may be about 135 cm fromthe sensor and the closest point may be about 5 cm. Such closest andfurthest points may be considered in reference to a measurement from thefinger tips. FIG. 12 illustrates the amplitude, frequency and phasecharacteristics that may be processed for detection of the gesture.

Shaving Motions

The system may be applied for many types of activity, preferablyassociated with repeating motions. Examples can include detecting andclassifying activities such as rinsing, combing, brushing (e.g., hair orteeth) or shaving strokes, etc. In some cases, the system may assumethat the primary motions recorded contain a particular activity (e.g.,shaving information and/or rinsing). Analysis of the gesture channel canpermit, for example, estimating total number of strokes, detecting thechange in direction of the motion may be determined. Similarly, relativedirection of stroke—up/down or down/other, etc. may be determined. Therelative direction of the motion source may be detected. Rate of strokemay be determined.

By detecting a likely stroke event, it is possible to calculate andprovide an estimate of the rate in strokes per minute. Peak high rateevents are marked as possible rinse events.

In some versions of the system an activity type or gesture processor mayimplement any one or more of the following Processing steps:

Calculate spectral content of gesture signal(s)

apply Fast Fourier transform and find peak (frequency domain) in arolling window

Calculate the distance between each sinusoidal-like peak

Calculate zero crossings of signal (time domain)

Estimate relative direction of movement and duration

Extract phase shift between the two channels.

Alternative time frequency analysis such as short time Fourier transformor wavelets may also be implemented.

In general, the complex sensor signal is based on arm movement, headmovement, torso movement etc. Other movements may also be detected.

In some versions, the clockwise/anti-clockwise direction changeinformation may be clocked to produce an impulse to represent a changein direction. These pulses may be implemented for a counter, and groupedinto different rates of occurrence. FIG. 13 illustrates the change indirection for detection as I/Q phase signal difference varies.

Therefore, typical rates consistent with the act of shaving can benoted, and thus the period of shaving deduced. An increase in rateassociated with excess high frequency information can be inferred as thearm moving to the face, or the rinsing of a razor blade.

An advantage of using an RF sensor for detecting of shaving or othermotions and gestures is the enhanced privacy versus say a video basedsystem that captures or processes pictures/video from a user or group ofusers.

A reduction in rate and direction change can be used to detectbreathing. In addition, time domain and frequency domain processing isapplied to the signals to localize specific bands.

Breathing can be further separated from confounding human body sway bydetecting a relative change in rate with an unexpected direction changebehaviour characteristic. FIG. 13 illustrates change in directiondetection as I/Q phase difference varies. In FIG. 13, the IQ plot in theupper left panel represents a trace moving in a clockwise direction, andupper left showing a trace moving in an anti-clockwise direction. Achange from a clockwise to anti-clockwise direction (or vice versa)gives rise to the direction change trace shown in the lower left andright panels by the top line therein. The middle and bottom linesrepresent either the I or Q channels respectively in this example.

In one example version, strokes of the activity, (e.g., shaving strokes)may be counted with application of processing that includes thefollowing

Band pass filtering

Calculating the change of state from clockwise/anticlockwise.

Applying hysteresis (avoid flipping state on small blips in signal,e.g., foldover).

Suppressing feature update around the zero point

Differentiating the resulting signal.

Counting the number of transitions (e.g., to identify a return stroke).

A signal graph illustrating such detection processing is shown in FIG.14.

In executing such gesture/activity detecting training, classificationmay be performed in the following manner as illustrated in FIG. 15. Oneset of recordings from the sensor may be accessed in a read step 1502,may be used as a training set. Suitable detection features (with phase,frequency and amplitude) may be produced, such as in a featuregeneration step 1504. In a training setup step 1506, a classifierconfiguration may be created for particular motions/gestures. Thefeatures may then be processed in a training classify step 1508 torelate a motion to the most relevant of the calculated features. Thetraining classifying may be repeated if further tuning is desired atcheck step 1512, such as if improved classification training is desired.In a pre-testing setup step 1505, a classifier configuration may beaccessed for evaluating features of previously classifiedmotions/gestures. These pre-classified motions may then be compared withnewly generated features in a classification step 1507 to identify oneof the pre-classified motions based on the features. Optionally, theperformance of the classifier from training or testing may be assessedin video performance step 1510, 1509 using the identified features tocompare with video based annotations (i.e., where a simultaneous videois recorded during performance of known gestures to act as a timestampreference for later annotation of the motion signals; this requireshuman scoring of the signals and/or a separate log of motion/gestureevents to be performed) and, based on the result of the comparison, thefeatures may need to be fine-tuned. An independent test set may then beused to test the resulting configuration of features. For this type ofsupervised learning (unsupervised learning is also possible using othertechniques), an independent test set is held back from the training setin order to check the likely real world performance of a system (i.e.,the performance on unknown data). During the development process,iteration is carried out on the training set in order to maximiseperformance, and aims to use the minimum number of features thatmaximise performance where possible. Principal Component Analysis (PCA)or other dimensionality reduction may be implemented in order to selectsuch features. It will be recognized that steps 1502, 1504, 1505 and1507 may be implemented by a processor or controller, in or associatedwith a detection device 100, for the purposes of making motionidentification as previously described, when not implementing trainingand testing.

For example, a Kolgomorov-Smirnov (KS) goodness-of-fit hypothesisstatistical test may be implemented to compare the cumulativedistribution function of the target block of data to the training data.Such a block by block classification is illustrated in the example ofFIG. 16. It may be implemented with any one or more of the followingprocesses:

(1) Biomotion Block Division

The I and Q signal data can be split up into either continuousnon-overlapping or partially overlapping blocks. For example, a blocklength of 1*160 samples (1 seconds at 160 Hz) with a 50% overlap couldbe used or some other combination. Computational complexity can betraded for precision by varying the block length and/or by varying theamount of overlap.

(2) Block Pre-Processing

The block of data may be checked to see if the data falls within apresence or absence section (i.e., is there a user with a breathing rateand/or heartbeat within range of the sensor or sensors; for breathingrate detection, between 15 and 60 seconds plus of data may be requiredto detect multiple breathing cycles). Furthermore, the block may bechecked to see that no possible RF interference signals are detected(i.e., to separate motion/gesture signals from strong sources of RFinterference that might be detected by an RF transceiver; also, othernon biomotion sources such as fans may be detected and rejected at thisstage). If the block under consideration does not meet these criteria,it may optionally not be classified further. The block or blocks mayalso be cross referenced and/or correlated with other informationsources of the user or the room environment, in order to check thelikelihood of a user actually being in the vicinity of the sensor; forexample, data from a wearable device, location or motion data from acell phone, room environmental sensors, home automation or othersecurity sensors.

(3) Feature Extraction

For the block under consideration, a number (either all or a subset) oftime-domain (temporal) and frequency domain or time/frequency featuresmay be calculated as follows. It is noted that different block lengthsmay be considered simultaneously.

transformed trimmed mean and median (said transformation being forexample, but not limited to, the square root, squared or log) of the I &Q signals (or of derived features)

transformed spread in the signals (said transformation being forexample, but not limited to, the square root, squared or log) calculatedusing interpolation or otherwise, covering a defined range (for example,but not limited to, the range from 5% to 95% or of interquartile range).

The envelope of the signal (I & Q) using a Hilbert transform

-   -   The relative amplitude of the signal (I&Q) to surrounding        examples of the signal    -   The zero crossings of the signal (I & Q)    -   The peak frequency in a moving window    -   The ratios of peak frequency to second and third harmonics    -   The phase direction (clockwise or anticlockwise)    -   The phase velocity    -   The existence (or lack thereof) of a breathing and/or cardiac        signal in the signal (i.e., relating the motion to a biomotion,        e.g., that motion made by a person)    -   The presence of a similar or difference in motion signal in I &        Q channels

(4) Block Classification

As an example, for an input feature set with a characteristicdistribution, the Kolgomorov Smirnov (KS) two sample non parametricgoodness of fit test may be used to compare this reference sample (e.g.,features of a shaving motion, particular hand gesture derived from time,frequency, phase etc.) to a new sample distribution that has beencaptured by the sensor(s) (e.g., quantifying a distance between theempirical distribution function of the new sample detected and thecumulative distribution function of the reference distribution). Amultivariate version of the KS may also be implemented, although thismay require multiple cumulate density function comparisons to be made.

As another example, a linear discriminant classifier (LDC), based onFisher's linear discriminant rule, is applied to each non-overlapped oroverlapped block. For each block of data fed in, there are multiplepredetermined output classes—e.g., different motion or gesture states.The classifier outputs a set of numbers representing the probabilityestimate of each class, in response to a set of input features. Lineardiscriminants partition the feature space into different classes using aset of hyper-planes. Optimisation of the model is achieved throughdirect calculation and is extremely fast relative to other models suchas neural networks.

The training of a LDC proceeds as follows. Let x be a dx 1 column vectorcontaining feature values calculated from a data set. We wish to assignx to one of c possible classes (c=2 in our case). A total of N featurevectors are available for training the classifier, with the number offeature vectors representing class k equal to N_(k), i.e.:

$\begin{matrix}{N = {\sum\limits_{k}N_{k}}} & (1)\end{matrix}$

The n^(th) training vector in class k is denoted as x_(k,n). Theclass-conditional mean vectors μ_(k) are defined as:

$\begin{matrix}{\mu_{k} = {\frac{1}{N}{\sum\limits_{n = 1}^{N_{k}}x_{k,n}}}} & (2)\end{matrix}$

We now define a common covariance matrix defined over all classes (i.e.,we assume that each class only differs in its mean value, and not in itshigher order statistics). The common covariance matrix is defined as:

$\begin{matrix}{\Sigma = {\frac{1}{N - c}{\sum\limits_{k = 1}^{c}{\sum\limits_{n = 1}^{N_{k}}{\left( {x_{k,n} - \mu_{k}} \right)\;\left( {x_{k,n} - \mu_{k}} \right)^{T}}}}}} & (3)\end{matrix}$

The μ_(k)'s and E are calculated using training data. Once these valueshave been calculated, a discriminant value y_(k) for an arbitrary datavector x can be calculated using:y _(k)=−½μ_(k) ^(T)Σ⁻¹μ_(k)=μ_(k) ^(T)Σ⁻¹ x+log(π_(k))  (4)

Where π_(k) is the a priori probability of the vector x being from classk. It is easy to convert the discriminant values to posteriorprobabilities using:

$\begin{matrix}{{p\left( {k❘x} \right)} = \frac{\exp\left( y_{k} \right)}{\sum\limits_{k = 1}^{c}{\exp\left( y_{k} \right)}}} & (5)\end{matrix}$

This formulation provides a mapping from discriminant value to posteriorprobabilities. The final class assigned to x is the class with thehighest posterior probability. This becomes the block output.

However, the system can also employ methods such as neural networks,deep learning analysis etc.—especially where reasonable computing poweris available. More complex methods including morphological signalprocessing (e.g., such as may be used in image processing) can augmentfeature analysis when using more complex classification methods; thesemay be more appropriate for detecting patterns seen in complexmotions/gestures.

The periodic nature of the activity is further illustrated in the signalgraph of FIG. 17, showing the I channel, the Q channel, the stroke andstroke rate for the activity. In this example assuming a shavingactivity, the fourth (lowest) axis depicts a probable razor rinse periodwith black dots (labelled “DD” in FIG. 17 in the lowest panel labelled“stroke rate”—the high rate areas indicate these rinse points). Thisclearly illustrates detection of the periodic nature of the shavingactivity.

Further Example Gestures/Movements

As further illustrated in FIGS. 18-25 additional motion gestures may bedetected by analysis of the phase, frequency and/or amplitude of thesensor gesture channel signals. Although certain distances from thesensor are provided, it will be recognized that these distances may bealtered depending on the configured detection range of the sensor.

Gesture 1:

Gesture 1 may be considered in reference to FIGS. 18A-C. In thisexample, the sensor may be positioned a distance (e.g., 70 cm) from thecentre of the chest. The sensor is spaced from the gesturing subject inthe direction of the viewer of FIG. 18A (this is also the case with thesubsequent FIGS. 19A, 20A, 21A, 22A, 23A and 24A). The furthest pointduring the gross motion may be approximately 55 cm from the sensor andthe closest point may be approximately 45 cm. The furthest point may bemeasured in reference to the finger tips. As shown in FIG. 18A, the handmovement is performed with the arm parallel to the sensor. Only the handmoves back and forth perpendicular to the sensor. The complete gesture 1takes approximately 2 seconds. The motion may be performed from asitting or standing position. As illustrated in FIGS. 18B (10repetitions of gesture) and 18C (single gesture) features of any one ormore of the phase, frequency and amplitude may be classified fordetection of the gesture or the repeated gesture.

Gesture 2:

Gesture 2 may be considered in reference to FIGS. 19A-C. In thisexample, the sensor was positioned approximately 70 cm from the centreof the chest. The gesture may be considered waving a hand in front ofthe sensor. The furthest point during the gross motion was approximately50 cm from the sensor and the closest point was approximately 45 cm. Thefurthest point was measured to the finger tips at an angle ofapproximately 24 degrees from the sensor. As illustrated in FIG. 19A,movement begins with the arm parallel to the sensor. The hand only movesback and forth, parallel to the sensor. The complete Gesture takes lessthan approximately 2 seconds. The motion may be performed whilestanding, lying or from in a sitting position.

As illustrated in FIGS. 19B (10 repetitions of gesture) and 19C (singlegesture) features of any one or more of the phase, frequency andamplitude may be classified for detection of the gesture or the repeatedgesture.

Gesture 3:

Gesture 3 may be considered in reference to FIGS. 20A-C. In thisexample, the sensor was positioned approximately 70 cm from the centreof the chest. The furthest point during the gross motion wasapproximately 85 cm from the sensor and the closest point was 45 cm. Thefurthest point is measured in reference to the finger tips. The closestpoint is the shortest distance from the sensor to arm, rather than thefinger tips. As illustrated in FIG. 20A, the arm and hand movementbegins with the arm parallel to the sensor. The arm is then crossed overthe body before returning to the original position. The complete gesturetakes approximately 2 seconds. The motion may be performed whilestanding, lying or from in a sitting position.

As illustrated in FIGS. 20B (10 repetitions of gesture) and 20C (singlegesture) features of any one or more of the phase, frequency andamplitude may be classified for detection of the gesture or the repeatedgesture.

Gesture 4:

Gesture 4 may be considered in reference to FIGS. 21A-C. In thisexample, the sensor was positioned approximately 70 cm from the centreof the chest. The furthest point during the gross motion wasapproximately 60 cm from the sensor and the closest point wasapproximately 45 cm. The furthest point is measured in reference to thefinger tips. The closest point is the shortest distance from the sensorto the arm, rather than the finger tips. As shown in FIG. 21A, the armand hand movement begins with the arm raised, with the finger tipspointing in an upward direction, in parallel to the sensor. The armmoves in parallel to the sensor. The complete gesture takes less thanapproximately 2 seconds. The motion may be performed while standing,lying or from in a sitting position.

As illustrated in FIGS. 21B (10 repetitions of gesture) and 21C (singlegesture) features of any one or more of the phase, frequency andamplitude may be classified for detection of the gesture or the repeatedgesture.

Gesture 5:

Gesture 5 may be considered in reference to FIGS. 22A-C. In thisexample, the sensor was positioned approximately 95 cm from the centreof the chest. The furthest point during the gross motion wasapproximately 135 cm from the sensor and the closest point wasapproximately 5 cm. The closest and furthest points are measured inreference to the finger tips. As shown in FIG. 22A, the movement beginswith the arm fully extended. The hand is then swung completely acrossthe body. The complete gesture takes less than approximately 4 seconds.The motion may be performed while standing, lying or from in a sittingposition.

As illustrated in FIGS. 22B (10 repetitions of gesture) and 22C (singlegesture) features of any one or more of the phase, frequency andamplitude may be classified for detection of the gesture or the repeatedgesture or the repeated gesture.

Gesture 6:

Gesture 6 may be considered in reference to FIGS. 23A-C. In thisexample, the sensor was positioned approximately 70 cm from the centreof the chest. The furthest point during the gross motion wasapproximately 95 cm from the sensor and the closest point wasapproximately 50 cm. The furthest point is measured in reference to thefinger tips. The closest point is the shortest distance from the sensorto the shoulder, rather than the finger tips. As shown in FIG. 23A, thearm and hand movement begins with the arm fully extended, above thehead. The hand is then swung down in a 90 degree angle. The completeGesture took approximately 3 seconds. The motion may be performed whilestanding, lying or from in a sitting position.

As illustrated in FIGS. 23B (10 repetitions of gesture) and 23C (singlegesture) features of any one or more of the phase, frequency andamplitude may be classified for detection of the gesture or the repeatedgesture.

Gesture 7:

Gesture 7 may be considered in reference to FIGS. 24A-C. In thisexample, the sensor was positioned approximately 70 cm from the centreof the chest. The furthest point during the gross motion wasapproximately 52 cm from the sensor and the closest point wasapproximately 50 cm. As shown in FIG. 24A, the arm and hand movementbegins with the arm parallel to the sensor and the palm of the handfacing upwards. The hand is then pulsed up approximately 15 cm beforereturning to the original position. The complete gesture tookapproximately 2 seconds. The motion may be performed while standing,lying or from in a sitting position.

As illustrated in FIGS. 24B (10 repetitions of gesture) and 24C (singlegesture) features of any one or more of the phase, frequency andamplitude may be classified for detection of the gesture or the repeatedgesture.

Rollover Movement 1

Rollover detection may be considered in reference to FIGS. 25A-B. Forsleep information detection, a rollover may be taken as an indicationthat the person is having difficulty sleeping. In this example, themovement begins with a person on their back, for example. The personrolls onto their side towards the sensor which may take approximately 2seconds. There may be a pause thereafter (such as about 1 second in thetest example). The person then rolls away from the sensor to the initialposition, which may take approximately 2 seconds. In the signal data ofthe figures, the complete movement takes 5 seconds (two rollovers). Thisis repeated 10 times in the data.

As illustrated in FIGS. 25A (10 repetitions of rollover motion) and 25B(rollover), features of any one or more of the phase, frequency andamplitude may be classified for detection of the motion or the repeatedmotion.

Rollover Movement 2

Rollover detection may be further considered in reference to FIGS.26A-B. In this example, the movement begins with the subject on theirback, for example. The person will then roll onto their side away fromthe sensor which may take approximately 2 seconds. There may be a pausethereafter (such as about 1 second in the text example). The person maythen roll back towards the sensor to the initial position. This may takeapproximately 2 seconds. In the signal data of the figures, the completemovement takes 5 seconds (two rollovers). This is repeated 10 times inthe data.

As illustrated in FIGS. 26A (10 repetitions of rollover motion) and 26B(rollover), features of any one or more of the phase, frequency andamplitude may be classified for detection of the motion or the repeatedmotion.

Rollover Movement 3

Rollover detection may be further considered in reference to FIGS.27A-B. In this example, the movement is a little longer than that of therollover of FIG. 26 (rollover movement 2). The movement begins with thesubject on their back, for example. The person will then roll onto theirfront away from the sensor which may take approximately 3 seconds. Theremay be a pause thereafter (such as about 1 second in the text example).The person may then roll back towards the sensor to initial position.This may take approximately 3 seconds. In the signal data of thefigures, the complete movement takes 7 seconds (two rollovers). This isrepeated 10 times in the data.

As illustrated in FIGS. 27A (10 repetitions of rollover motion) and 27B(rollover), features of any one or more of the phase, frequency andamplitude may be classified for detection of the motion or the repeatedmotion.

Rollover Movement 4

Rollover detection may be further considered in reference to FIGS.28A-B. In this example, the movement is a little longer than that of therollover of FIG. 25 (rollover movement 1). The movement begins with thesubject on their back, for example. The person will then roll onto theirfront toward the sensor which may take approximately 3 seconds. Theremay be a pause thereafter (such as about 1 second in the text example).The person may then roll back away from the sensor to the initialposition. This may take approximately 3 seconds. In the signal data ofthe figures, the complete movement takes 7 seconds (two rollovers). Thisis repeated 10 times in the data. As illustrated in FIGS. 27A (10repetitions of rollover motion) and 27B (rollover), features of any oneor more of the phase, frequency and amplitude may be classified fordetection of the motion or the repeated motion.

In one alternative approach, a global feature may be extracted directlyfrom the spectrogram in order to provide a reference signature. Asdepicted in the gesture and rollover figures, a characteristic patternfor each gesture can be seen in the colour spectrogram. Such an approachmay be performed by processing or analysing the colour information inthe spectrogram pixels—e.g., in a block by block or region approach.Optionally, enhancement may be performed, including edge detection andenclosing specific patterns; this can be effective to remove or reducenoise in the surrounding pixels. The colour may be processed in, forexample, RBG (red green blue) or CMYK (cyan magenta yellow black)depending on the colour space; each may be treated as a separatechannel. Colour intensities can be separated by intensity value (e.g.,low, low-medium, medium, medium-high, high or some other combination),and then passed into a classifier, such as a neural network. Forexample, consider FIG. 22C and the colour spectrogram and the processingimages of FIG. 30. Edge enhancement here may be directed at capturingthe outline of the red blobs, and rejecting the stippled blue/purpleregion. The shape of the red region with yellow streaks thus provides aninitial signature (template) for this gesture type, and can be used in asupervised learning classifier. The variability of multiple iterationsof this gesture (movement) is shown in FIG. 22B, although the same basicshape and colour persists. This pattern can be averaged from thisrepeated movement, and provide a training input for this target gesture(movement).

FIG. 30 shows images. From top to bottom panel: the top panel containsRGB colour space channels, the second panel depicts R (red) channelonly, the third depicts G (green), the fourth B (blue) and the bottomB-D is the blue channel with blob detection applied to the intensityvalues, and shifting slightly to the left to remove the frequencycomponent at the very left (shown in panels 1-4). The maximum frequencyranges from 170 Hz (rightmost “blob”) to 210 Hz (leftmost “blob”) Thus,as illustrated in reference to FIG. 30 (second image up from thebottom), the image data may be processed to split (see FIG. 22[C] fromthe original (top)) the colour data of the top image RGB into any one ormore of the discrete red, green, blue color channels (top-middle image R(red), middle image G (green) and bottom two images B (blue)respectively) channels and selecting main blob areas. To the human eye,clearest signature is evident in the Blue channel (bottom); i.e.,consider the black region (ignoring the vertical stripe to the left ofthe image). The bottom image B-D illustrates overlaid blob detection ofthe isolated blue channel. Such splitting/color separation and blobdetection may be performed by suitable algorithm(s) of one or moreprocessors of the system, such as part of a process involving featuredetection and/or classification as described in more detail herein.

Multiple RF Sensors (e.g., Stereo Sensor System):

For an exemplar single RF sensor, the I and Q phase can be detected asthe user moves towards or away from it. Movement perpendicular to asingle sensor (across the face of the sensor) may have a much smallerrelative phase change (e.g., a movement in an arc across the sensor'sdetection plane will have a very low or no phase change measurable). Itis possible that additional sensors (e.g., a system with a second (andsubsequent) sensor(s) placed adjacent to the first sensor (e.g., at anangle from each other)) can be employed to also detect signals ofobjects moving in and out. For example, a second sensor, may bepositioned in the arc of the first sensor (e.g., the sensor might be at45 degrees to the first sensor or orthogonal (at 90 degrees) or otherappropriate differentiated angle with respect to first sensor). Thus,the effective stereo sensor system may more efficiently detect andcharacterise movement across various detection planes corresponding tothe sensors (e.g., a movement perpendicular to the first sensor may bemore clearly characterized by analysis of the signal of the secondsensor). In such a case, the movement/gesture classification may takeinto account the signal information from both sensors (e.g., featuresderived from the phase output from both sensors). Such a system may forexample return a different control signal based on the direction ofmotion in this manner. For a shaving analysis, the quadrant of the face(or other part of the body) could be determined. For a gamingimplementation, a specific localised movement could be determined.

Thus, two sensors can work cooperatively as a “stereo” system to detectand recognize gestures in two dimensions (2D), and three sensors can beused for identifying three dimensional characteristics (3D) of a gesturemovement, using for example, range gated RF sensors. Thus, a singlegesture may be characterized by obtaining detection signals frommultiple sensors. For the two sensor 2-dimension case (i.e., 2D), theI/Q signals from each of a sensor1 and sensor2 (differential in I1, I2,Q1, Q2 for two sensor case—left and right), can be analyzed by aprocessor. The resulting difference in amplitude and phase provides an“x”, “y” output. In some cases, three sensors may be implemented in acooperative system to add the “z” axis, in order to provide fine grainedthree-dimensional gesture recognition data in the resulting sensorfield. In such as case, the differentials of I1, I2, I3, Q1, Q2, Q3 maybe evaluated by a processor with the signals from the three sensor caseto discriminate a single gesture. In some embodiments, a maximum phasemay be obtained by placing at least two of the three sensors to beorthogonal to each other.

In some versions, a multi-ray antenna (phase array antenna) may beimplemented on one sensor if the antennas are separated. This mayeliminate the need for a second sensor.

In some cases, the RF sensor motion signals may, for example, beup-converted to audio frequencies such that a gesture may be detected(recognised) by a specific audio signature (e.g., by producing a soundfrom a speaker with the upconverted signals). This could allow a humanto distinguish different types of gesture, or to utilise known audiorecognition approaches to augment the classification and/or devicetraining.

In this specification, the word “comprising” is to be understood in its“open” sense, that is, in the sense of “including”, and thus not limitedto its “closed” sense, that is the sense of “consisting only of”. Acorresponding meaning is to be attributed to the corresponding words“comprise”, “comprised” and “comprises” where they appear.

While particular embodiments of this technology have been described, itwill be evident to those skilled in the art that the present technologymay be embodied in other specific forms without departing from theessential characteristics thereof. The present embodiments and examplesare therefore to be considered in all respects as illustrative and notrestrictive. For example, whilst the disclosure has described thedetection of movements such as hand/arm based gestures and roll-overs,the same principal is applicable to other large scale motions, such asuser moving between a lying and a sitting position in bed (and viceversa), reaching for a specific target (a table lamp, or a respiratoryapparatus) etc.

It will further be understood that any reference herein to subjectmatter known in the field does not, unless the contrary indicationappears, constitute an admission that such subject matter is commonlyknown by those skilled in the art to which the present technologyrelates.

PARTS LIST

-   -   Detection apparatus 100    -   Read step 1502    -   Feature Generation step 1504    -   Pre-testing setup step 1505    -   Training setup step 1506    -   Classification step 1507    -   Training classify step 1508    -   Video performance step 1509    -   Video performance step 1510    -   Check step 1512

The invention claimed is:
 1. A radio frequency motion sensing apparatuscomprising: a radio frequency transmitter configured to emit radiofrequency signals; a receiver configured to receive reflected ones ofthe emitted radio frequency signals, a motion channel circuit configuredto process the received reflected ones of the emitted radio frequencysignals and produce motion output signals; and a processor configured toevaluate the motion output signals and identify a motion based on phaseof the motion output signals, wherein the processor is configured toevaluate the motion output signals and identify the motion by evaluationof variation in a phase difference between I and Q signals, wherein theprocessor is configured to generate a direction change signal from thevariation, the direction change signal representing a plurality ofdirection changes from the variation in phase difference.
 2. Theapparatus of claim 1 wherein the identified motion comprises at leastone of hand gesture, an arm gesture or a combined hand and arm gesture.3. The apparatus of claim 1 wherein the identified motion comprises arollover motion.
 4. The apparatus of claim 1 wherein the identifiedmotion comprises an activity.
 5. The apparatus of claim 4 wherein theidentified motion comprises a shaving activity.
 6. The apparatus ofclaim 1 wherein the motion output signals comprise in phase andquadrature phase signals.
 7. The apparatus of claim 1 wherein theemitted radio frequency signals comprise pulsed radio frequencyoscillating signals.
 8. The apparatus of claim 1 wherein the motionchannel circuit comprises a bandpass filter.
 9. The apparatus of claim 1wherein the apparatus demodulates the received reflected ones of theemitted radio frequency signals with signals representing the emittedradio frequency signals.
 10. The apparatus of claim 1 wherein theapparatus calculates time difference between the emitted radio frequencysignals and the received reflected ones of the emitted radio frequencysignals and identifies the motion based on the calculated timedifference.
 11. The apparatus of claim 1 wherein the motion channelcircuit comprises an antialiasing filter.
 12. The apparatus of claim 1wherein the processor is configured to classify a motion based on aplurality of features calculated from any two of amplitude, phase andfrequency of the motion output signals.
 13. The apparatus of claim 1wherein the processor is configured to classify or identify a motionbased on a duration calculated with any one or more of amplitude, phaseand frequency of the motion output signals.
 14. The apparatus of claim12 wherein the processor is configured to calculate the plurality offeatures from each of the amplitude, phase and frequency of the motionoutput signals.
 15. The apparatus of claim 12 wherein the plurality offeatures comprise a determined duration derived from analysis of any oneor more of the amplitude, phase and frequency of the motion outputsignals.
 16. The apparatus of claim 15 wherein the calculated pluralityof features comprise one or more of: (a) a frequency characteristicderived from stopped frequency through a gesture in motion up to somemaximum frequency, then back to stopped again; (b) a time and frequencyanalysis of the motion output signals comprising any of short timeFourier transform, peak and harmonic tracking and/or channel processingof an I and/or Q channel(s); (c) a phase characteristic comprising anyof: a phase difference between I and Q signals and an evaluation of arepetitive signal within a certain number of standard deviations of amean of characteristic change; (d) an amplitude characteristiccomprising any of: peak and trough detection, zero crossing detection,and envelope of signal detection; and (e) a skewness, kurtosis, spreadin frequency, phase, amplitude, mean, and/or standard deviation.
 17. Theapparatus of claim 14 wherein the processor is configured to compare thecalculated plurality of features to one or more thresholds.
 18. Theapparatus of claim 1 wherein the processor is configured to identify themotion by selecting one from a plurality of predetermined motions. 19.The apparatus of claim 1 wherein the processor is configured to count anumber of occurrences of the identified motion.
 20. The apparatus ofclaim 1 wherein the processor is further configured to generate acontrol signal for operation of a device based on the identified motion.21. The apparatus of claim 1 wherein the processor is further configuredto generate different control signals for different operations of adevice based on different identified motions.
 22. The apparatus of claim1 wherein the processor is configured to evaluate the motion outputsignals and identify a motion based on motion output signals from aplurality of sensors.
 23. The apparatus of claim 22 wherein theplurality of sensors comprises a first sensor and a second sensor,wherein the processor evaluates I and Q signals from the first sensorand the second sensor to identify the motion.
 24. The apparatus of claim23 wherein the processor determines an I differential and a Qdifferential of the I and Q signals from the first sensor and the secondsensor.
 25. The apparatus of claim 22 wherein the plurality of sensorscomprises a first sensor, second sensor and a third sensor, wherein theprocessor evaluates I and Q signals from the first sensor, the secondsensor and the third sensor to identify the motion.
 26. The apparatus ofclaim 25 wherein at least two of the first sensor, second sensor andthird sensor are positioned to be orthogonal to each other.
 27. Theapparatus of claim 23, wherein the evaluated I and Q signals are used bythe processor to identify movement characteristics in more than onedimension.
 28. The apparatus of claim 1, wherein the processor isconfigured to extract one or more of the following: velocity, change invelocity, distance, change in distance, direction and change indirection.
 29. A method for radio frequency motion sensing comprising:with a radio frequency transmitter, emitting radio frequency signals;with a receiver, receiving reflected ones of the emitted radio frequencysignals, processing the received reflected ones of the emitted radiofrequency signals to produce motion output signals with a motion channelcircuit; and in a processor, evaluating the motion output signals andidentifying a motion based on phase of the motion output signals,wherein the evaluating the motion output signals and identifying themotion comprises evaluation of variation in a phase difference between Iand Q signals, wherein the processor is configured to generate adirection change signal from the variation, the direction change signalrepresenting a plurality of direction changes detected from thevariation in phase difference.
 30. The method of claim 29 wherein theidentified motion comprises any one of hand gesture, an arm gesture anda combined hand and arm gesture.
 31. The method of claim 29 wherein theidentified motion comprises a rollover motion.
 32. The method of claim29 wherein the identified motion comprises an activity.
 33. The methodof claim 32 wherein the identified motion comprises a shaving activity.34. The method of claim 29 wherein the motion output signals comprise inphase and quadrature phase signals.
 35. The method of claim 29 whereinthe emitted radio frequency signals comprise pulsed radio frequencyoscillating signals.
 36. The method of claim 29 wherein the motionchannel circuit comprises a bandpass filter.
 37. The method of claim 29further comprising demodulating the received reflected ones of theemitted radio frequency signals with signals representing the emittedradio frequency signals.
 38. The method of claim 29 further comprisingcalculating time difference between the emitted radio frequency signalsand the received reflected ones of the radio frequency signals andidentifying the motion based on the calculated time difference.
 39. Themethod of claim 29 wherein the motion channel circuit comprises anantialiasing filter.
 40. The method of claim 29, further comprising,with the processor classifying a motion based on a plurality of featurescalculated from any two of amplitude, phase and frequency of the motionoutput signals.
 41. The method of claim 29 wherein the processorclassifies or identifies a motion based on a duration calculated withany one or more of amplitude, phase and frequency of the motion outputsignals.
 42. The method of claim 40 further comprising calculating inthe processor the plurality of features from each of the amplitude,phase and frequency of the motion output signals.
 43. The method ofclaim 40 wherein the plurality of features comprise a determinedduration derived from analysis of any one or more of the amplitude,phase and frequency of the motion output signals.
 44. The method ofclaim 42 wherein the calculated plurality of features comprise one ormore of: (a) a frequency characteristic derived from stopped frequencythrough a gesture in motion up to some maximum frequency, then back tostopped again; (b) a time and frequency analysis of the motion outputsignals including any of a short time Fourier transform, peak andharmonic tracking, and processing of I and/or Q channel(s); (c) a phasecharacteristic including any of a phase difference between I and Qsignals or an evaluation of a repetitive signal within a certain numberof standard deviations of a mean of characteristic change; (d) anamplitude characteristic including any of a peak and trough detection, azero crossing detection, an envelope of signal detection; and (e) askewness, kurtosis, spread in frequency, phase, amplitude, mean, and/orstandard deviation.
 45. The method of claim 42 further comprising, inthe processor, comparing the calculated plurality of features to one ormore thresholds.
 46. The method of claim 29, further comprising, in theprocessor, identifying the motion by selecting one from a plurality ofpredetermined motions.
 47. The method of claim 29 further comprising, inthe processor, counting a number of occurrences of the identifiedmotion.
 48. The method of claim 29 further comprising, with theprocessor, generating a control signal for operation of a device basedon the identified motion.
 49. The method of claim 29 further comprisingwith the processor generating different control signals for differentoperations of a device based on different identified motions.
 50. Themethod of claim 29 wherein the processor evaluates the motion outputsignals from a plurality of sensors and identifies a motion based on theevaluated motion output signals.
 51. The method of claim 50 wherein theplurality of sensors comprises a first sensor and a second sensor,wherein the processor evaluates I and Q signals from the first sensorand the second sensor to identify the motion.
 52. The method of claim 51wherein the processor determines an I differential and a Q differentialof the I and Q signals from the first sensor and the second sensor. 53.The method of claim 50 wherein the plurality of sensors comprises afirst sensor, a second sensor and a third sensor, wherein the processorevaluates I and Q signals from the first sensor, the second sensor andthe third sensor to identify the motion.
 54. The method of claim 53wherein at least two of the first sensor, the second sensor and thethird sensor are positioned to be orthogonal to each other.
 55. Themethod of claim 49, wherein the evaluated I and Q signals are used bythe processor to identify movement characteristics in more than onedimension.
 56. The method of claim 29, the method comprising extracting,with the processor, one or more of the following: velocity, change invelocity, distance, change in distance, direction and change indirection.
 57. The method of claim 29 wherein the evaluating the motionoutput signals and identifying a motion is based on frequency of themotion output signals.
 58. The method of claim 29 wherein the evaluatingthe motion output signals and identifying a motion is based on amplitudeof the motion output signals.
 59. The method of claim 29 wherein radiofrequency signals emitted are pulsed continuous wave radio frequencysignals.
 60. The method of claim 29 wherein the motion output signalsare produced from demodulating the received reflected ones of the radiofrequency signals by the emitted radio frequency signals.
 61. The methodof claim 29 further comprising range gating of the radio frequencysignals.
 62. The method of claim 29 wherein the evaluating the motionoutput signals and identifying the motion based on the phase differencebetween I and Q signals comprises evaluation of a phasor determined fromthe I and Q signals.
 63. The method of claim 29 wherein evaluating themotion output signals and identifying the motion is based on a phasedifference between the emitted radio frequency signals and the receivedreflected ones of the radio frequency signals.
 64. The method of claim29 wherein further comprising detecting physiological characteristics ofa user.
 65. The apparatus of claim 1 wherein the processor is configuredto evaluate the motion output signals and identify the motion based onfrequency of the motion output signals.
 66. The apparatus of claim 1wherein the processor is configured to evaluate the motion outputsignals and identify the motion based on amplitude of the motion outputsignals.
 67. The apparatus of claim 1 wherein radio frequency signalsemitted are pulsed continuous wave radio frequency signals.
 68. Theapparatus of claim 1 wherein the motion output signals are produced fromdemodulating the received reflected ones of the radio frequency signalsby the emitted radio frequency signals.
 69. The apparatus of claim 1wherein the radio frequency motion sensing apparatus is configured torange gate the radio frequency signals.
 70. The apparatus of claim 1wherein the processor is configured to evaluate the motion outputsignals and identify the motion based on the phase difference between Iand Q signals by evaluation of a phasor determined from the I and Qsignals.
 71. The apparatus of claim 1 wherein the processor isconfigured to evaluate the motion output signals and identify the motionbased on a phase difference between the emitted radio frequency signalsand the received reflected ones of the radio frequency signals.
 72. Theapparatus of claim 1 wherein the radio frequency motion sensingapparatus is further configured to detect physiological characteristicsof a user.
 73. The apparatus of claim 1 wherein the processor is furtherconfigured to generate the direction change signal by applyinghysteresis to avoid a flipping state on fold-over.
 74. The apparatus ofclaim 73 wherein the processor is further configured to identify themotion by differentiating the direction change signal.
 75. The method ofclaim 29 wherein the processor generates the direction change signal byapplying hysteresis to avoid a flipping state on fold-over.
 76. Themethod of claim 75 wherein the processor identifies the motion bydifferentiating the direction change signal.